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Abstract 

We  describe  an  algorithm  for  the  0-1  knapsack  problem  (KI’)>  which 
relies  mainly  on  three  new  ideas.  The  first  one  is  to  focus  on  wlmL  we 
call  the  core  ol  the  problem,  namely  a knapsack  problem  equivalent  to  (KP)  , 
defined  on  a particular  subset  of  the  variables.  The  si ze  of  this  core  is 
usually  a small  fraction  of  the  full  problem  size,  and  does  not  seem  to 
increase  with  the  latter.  While  the  core  cannot  be  identified  without  solving 
(KP) , a satisfactory  approximation  can  be  found  by  solving  (LKP) , the  associated 
linear  program.  The  second  new  ingredient  is  a binary-search-type  procedure 
for  solving  (LKP)  which,  unlike  earlier  methods,  does  not  require  any  ordering 
of  the  variables.  The  computational  effort  involved  in  this  procedure  is 
linear  in  the  number  of  variables.  Finally,  the  third  new  feature  is  a simple- 
minded  heuristic  whose  accuracy  under  certain  conditions  grows  exponentially 
with  the  problem  size.  Computational  experience  wit'  gorithm  based  on  the 

above  ideas,  on  200  randomly  generated  test  problems  with  1,000-10,000  variables 
and  with  coefficients  ranging  from  between  10-100  to  between  10-10,000,  indicates 
that  for  such  problems  the  computational  effort  grows  linearly  with  the 
number  of  variables  and  logarithmically  with  the  range  of  coefficients. 

Total  time  for  the  200  problems  was  160  UNIVAC  1108  seconds,  and  the 
maximum  time  for  any  single  problem  was  3 seconds. 


SOLVING  LARGE  ZERO-ONE  KNAPSACK  PROBLEMS 


by 

Egon  11/tlus  and  Ititan  Zomc  1 
1.  Introduction 

We  consider  the  0-1  knapsack  problem 
(KP)  max  [cx  | ax  < a^,  x = 0 or  1,  j e N]  , 

where  c = (cj)  a°d  a = (a^)  are  positive  n-vectors  with  Integer  components, 
a is  a positive  integer,  and  N = {l,...,n]  . We  will  also  refer  to  the 
linear  program  associated  with  (KP), 

(LKP)  max  {cx  | ax  < a^,  0 < x < e]  , 

where  e = (1,...,1)  has  n components.  Our  approach  can  be  extended  in  a 
straight forward  manner  to  the  general  bounded-variables  (as  opposed  to  0-1) 
knapsack  problem. 

Several  fast  algorithms  are  available  for  the  solution  of  (KP)  (see, 
for  instance,  [41,  [10],  1 13 1),  based  on  firsL  solving  (LKP)  and  generating 
an  integer  solution  to  define  upper  and  lower  bounds,  then  using  these  bounds 
with  some  logical  tests  to  fix  as  many  variables  as  possible,  and  finally 
solving  the  knapsack  problem  in  the  remaining  variables  by  some  specialized 
version  of  implicit  enumeration. 

The  starting  point  of  each  of  these  algorithms  is  the  ordering  of  the 
variables  according  to  decreasing  cost-to-weight  (c.^/a^)  ratios,  which  is  the 
basis  of  the  method  used  for  solving  (LKP).  This  preprocessing  .if  the 
variables,  though  often  done  by  a separate  sorting  routine  and  therefore  (?) 
not  included  in  the  total  times  reported,  usually  constitutes  the  lion's 
share  of  the  computational  effort  required  to  solve  (KP). 
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Wc  propose  an  approach  to  the  0-1  knapsack  problem  which,  while  using 
in  its  final  stage  some  of  the  above  mentioned  techniques,  relies  mainly  on 
three  new  ideas.  The  first  one  is  to  focus  on  what  we  call  the  core  problem, 
i.e.,  the  subproblem  in  those  variables,  whose  cost/weight  ratio  falls  between 
the  maximum  and  the  minimum  c^/a^  ratio  for  which  x^  has  a different  value 
in  an  optimal  solution  to  (KP)  from  that  in  an  optimal  solution  to  (LKP)  (see 
section  2).  The  size  of  this  core  is  usually  a surprisingly  small  fraction 
of  the  full  problem  size,  and  does  not  increase  with  the  latter.  While 
precise  identification  of  the  core  would  require  the  solving  of  (KP) , 
a satisfactory  approximation  can  be  found  by  solving  (LKP), 

The  second  new  ingredient  is  a binary-search-type  method  for  solving 
(LKP)  without  ordering  the  variables  (see  section  3).  As  a byproduct, 
this  procedure  also  yields  a convenient  approximation  (IP)  to  the  core 
problem.  The  classical  procedure  for  solving  (LKP)  is  based  on  ordering 
the  variables  according  to  their  cost/weight  ratios,  an  operation  which 
by  itself  is  of  complexity  0(nlog2n) . The  computational  complexity  of 
our  procedure  is  0(n). 

The  third  idea  is  to  make  use  of  the  fact  that,  if  the  cost-to-weight 
ratios  of  the  approximate  core  problem  are  close  enough  to  each  other  (a 
condition  that  (IP)  tends  to  satisfy  by  construction),  there  exists  a simple 

heuristic  which  finds  an  optimal  solution  to  (IP)  with  a probability 
that  grows  exponentially  with  the  size  of  (IP)  (6ee  section  A). 

Procedures  based  on  the  above  three  ideas  usually  produce  an  integer 
solution  which  is  either  optimal,  or  provides  a sufficiently  good  lower  bound 
to  make  it  possible  to  fix  the  bulk  of  the  variables  outside  the  approximate 
core  problem.  This  is  done  by  a slightly  improved  version  of  some  logical 
tests  from  the  literature,  and  is  followed  by  implicit  enumeration  on  the 
remaining  variables  (see  section  5). 
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Computational  experience  on  200  randomly  generated  test  problems  with 
1000-10000  variables,  and  with  coefficients  ranging  from  10-  10000,  discussed  in 
section  6,  indicates  that  for  such  problems  the  computational  effort  (about  160 
UNIVAC  1108  seconds  for  the  200  problems,  the  maximum  time  for  any  single  problem 
being  3 seconds ) grows  linearly  with  the  number  of  variab leu , and  logarithmically 
with  the  range  of  coei f lcients . This  makes  it  possible  to  solve  randomly 
generated  0-1  knapsack  problems  of  virtually  any  size,  by  a computational 
effort  which  (for  large  enough  problems)  is  smaller  than  the  effort  required 
to  order  the  variables.  Problems  whose  cost-to-weight  ratios  are  all  within 
a small  interval  (or  are  all  equal),  which  have  been  found  difficult  for 
most  other  methods,  are  easily  handled  by  our  approach. 

On  the  other  hand,  there  is  a class  of  0-1  knapsack  problems  that  is 
hard  for  this  method  (as  well  as  for  others).  In  section  7 we  identify  this 
class  of  problems,  and  outline  a cutting  plane  method  which,  while  unable 
to  compete  with  the  above  approach  on  randomly  generated  problems,  is  often 
superior  on  problems  in  the  "hard"  category. 

2.  The  Core  Problem  and  Its  Approximations 

In  order  to  define  the  core  problem  mentioned  in  section  1,  we  will 
assume  N to  be  ordered  so  that 


for  an  optimal  solution  to  (KP),  define 

j*  = min  lj  e N | Xj  = 0} 

J**  = t J e N | Xj  = i}  , 

and 

- min  , j2  = max  tJ*.J**)  • 


Note  that  j.  = j*  except  for  the  case  when  x,  with  its  components  ordered 
as  specified  above,  is  of  the  form  ( 1 j • • • , 1 , 0, # • • , 0) . 

If  (KP)  has  more  than  one  optimal  solution.  Let  x be  one  which 
minimizes  the  size  of  the  interval  C = [J^jjjl.  We  will  call  C the  core  of  N, 
and  the  knapsack  problem 


(CP)  max  { E ex  | E a x < a_  - E a.,  x * 0 or  1,  JtC) 

JeC  J J jeC  J J j=l  J J 

the  core  of  (KP),  or  the  core  problem  associated  with  (KP). 

The  core  problem  is  easily  seen  to  be  equivalent  to  (KP)  in  the  following 


Proposition 


• I C | A 

_1.  x e R1  1 is  optimal  for  (CP)  if  and  only  if  x is  optimal 


for  (KP),  where 


j e C 

1 j < j, 

0 j > j. 


Proof.  If  x is  a solution  to  (CP)  better  than  x,  then  x/,  defined 
relative  to  x in  the  same  way  as  x is  defined  relative  to  x,  is  a solution 
to  (KP)  better  than  x,  a contradiction.  Conversely,  if  CKP)  has  a solution 
better  than  x,  its  projection  on  the  subspace  corresponding  to  C is  a solution 
to  (CP)  better  than  x,  again  a contradiction.  || 

Another  essential  property  of  the  core  is  the  following. 

Proposition  2.  (LKP)  has  an  optimal  solution  x such  that  x^  - 0,  Vj  < j ^ , 
and  Xj  = 1,  Vj  > J2- 

Proof . Let  x be  the  solution  to  (KP)  used  to  define  j^  and  With  N 

ordered  according  to  decreasing  c^/a^  ratios,  (LKP)  has  an  optimal  solution 
* of  ( tie  form  Kj  - I,  | t , 0 ■ s(  - I , x - 0,  j ^ f , i or  some  fcN.  If 
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x£  = 0,  then  x is  also  an  optimal  solution  to  (LKP)  and  the 
proposition  holds.  Now  let  0 < x£  <s  1.  If  f < J , then  cx  cx,  wliidi 
contradicts  the  optimality  of  x for  (LKP)*  If  f > j , then  x defined  by 
Xj  = 1,  j = l,...,j2>  Xj  = 0,  j > j2  is  a feasible  solution  to  (KP),  with 
cx  > cx,  which  contradicts  the  optimality  of  x for  (KP).  || 


For  randomly  generated  0-1  knapsack  problems  the  size  of  the  core  is 
usually  a small  fraction  of  the  size  of  the  full  problem,  and  does  not  seem  to 
depend  on  the  latter  (25  is  as  typical  a core  size  for  100-variable  problems 
as  it  is  for  problems  with  10,000  variables).  Since  the  core,  as  defined  above, 
cannot  be  identified  without  knowing  the  optimal  solutions  of  (KP),  an  idea 
which  suggests  itself  In  a natural  way  is  to  try  to  approximate  the  core.  Any 
interval  1CN  (where  N is  ordered  as  above)  containing  the  index  of  the  frac- 
tional component  of  an  optimal  solution  to  (LKP)  can  be  viewed  as  an  approximation 
to  the  core.  We  will  denote  by  (IP)  the  approximate  core  problem  defined  by 
I — [i^,i2l,  i.e.. 


(IP) 


max 


{ £ 

iel 


c.x. 
i i 


iel 


aixi 


V1 

< a0  " 2 ai 

i=l  1 


t - 0 or  1,  itl) 


The  choice  of  I involves  a tradeoff  between  the  probability  that  I-’C  (i.e., 
that  the  resulting  (IP)  is  equivalent  to  (KP)),  and  the  degree  of  difficulty 
of  solving  (IP). 

At  first  glance,  it  would  seem  that  approximating  the  core  requires  that 
we  solve  (LKP)  by  ordering  N according  to  decreasing  c£/a£  ratios.  In  the 
next  section  we  give  a procedure  which  finds  an  optimal  solution  x to  (LKP) 
without  ordering  N,  and  as  a byproduct  yields  a reasonably  sized  (for  randomly 
generated  problems)  interval  I = [i£,i2],  I^N,  containing  the  index  of  the 


f 


fractional  component  of  x 


3.  Solving  (LKP)  Without  Sorting 


It  is  well  known  that  if  N is  ordered  according  to 
ratios,  then  an  optimal  basic  solution  x to  (LKP)  is  of 


decreasing  c^/a^ 
the  form 


x 


j 


f-1 


2 

j-1 


j 


j < f 
j > f 

j - f 


where  f is  the  smallest  j eN  such  that  2 a > a . Hie  classical  method 

i=l 

of  solving  (LKP)  is  based  on  this  observation.  Since  sorting  n elements 
requires  by  itself  at  least  cnlog^n  comparisons  (c  a constant)  (see  [1|, 
Corollar'  heorem  3.4),  solving  (LKP)  by  the  classical  procedure  requires 
at  le  n't  operations. 

Ae  ordering  of  N is  not  unique,  neither  is  the  index  f. 

However,  the  ratio  c^/a^.  is  unique,  and  will  henceforth  be  called  the 
critical  ratio.  For  any  positive  scalar  X,  denote 


S.(X)  = 2 a S (X)  «=  S (X)  + 2 a 

i|(ci/ai)>X  i|(c./a1)=X 

'/c 

Then  the  critical  ratio  X = c^/a^  is  obviously  the  unique  solution  to  the 
pair  of  inequalities 

SX(X)  < aQ  < S2(X)  , 

and  solving  (LKP)  essentially  amounts  to  finding  the  critical  ratio  X . 

The  algorithm  below  (Procedure  1)  accomplishes  this  without  sorting 
the  variables.  In  its  statement,  the  ordering  of  N is  arbitrary.  At  each 
iteration,  Nq,  and  Np  denote  the  index  sets  of  the  variables  fixed  at  0, 


fixed  at  1,  and  free,  respectively.  The  sums  S^(X)  and  S2(X)  are  always 
defined  on  the  current  set  of  free  variables. 


Procedure  1. 

0.  Initialize:  set  NQ  = 0,  N^  = 0,  Np  = N,  and  go  to  1. 

1.  Choose  X e {c./a^}pe  , partition  Np  into 

T 


N>  = [i  e Np,  | (ci/ai)  > X} 


Nv  = [i  eNp  | (c1/a1)  < X] 


N = t1  e Np  | (Cp/a^  a X]  > 
and  calculate  S,(X),  S„(X).  Co  to  2. 


2.  If  S ^ (X ) < aQ  < S2(X),  stop:  X ' 

obtained  by  setting 


X.  An  optimal  solution  x to  (LKP)  is 

j tNj^  LUT’ 
j e NQ  U 


and  then  "filling  the  knapsack"  with  variables  x j , j eN  (any,  possibly 
including  one  at  a fractional  value). 

If  SjCX)  > a0,  i.e. , if  X is  too  small,  set  «-  N()  U N U N , and  Np  •-  M . 

If  S2(X)  < aQ,  i.e.,  if  X is  too  large,  set  ♦-  UN  UN-,  and  Np  «-  N . 

Then  go  to  3. 

3.  If  Np  > 0 (where  9 is  a fixed  threshold  value),  go  to  1.  Otherwise 

solve  the  linear  program  defined  on  Np  by  the  usual  method  (i.e.,  by 

ordering  N_,)  and  extend  the  solution  to  (LKP)  by  setting  x.  = 1,  j (N.  , 

* J 1 

Xj  = 0,  j « Nq.  This  solution  is  optimal. 
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The  threshold  value  8 in  step  3 is  meant  to  identify  the  problem  size 
under  which  the  usual  solution  method  (based  on  sorting  the  variables) 
becomes  preferable  to  the  one  consisting  of  steps  1-2.  We  have  found  0 = 25 
an  adequate  value. 

As  for  the  value  of  X in  step  1,  the  ideal  choice  would  of  course  be 
the  median  of  the  ratios  c^/a^,  ieN^.  Calculating  the  median  involves, 
however,  a certain  computational  effort.  Instead,  we  have  tried  the  following 

1 ^ i ^ j 

two  rules:  (a)  choose  X to  be  the  ratio  c./a,  closest  to  — (max 1-  min  — ); 

1 1 2 'l eN  ai  ieN  ai ' 

1 F 

(b)  choose  X to  be  the  median  of  the  first  k ratios  c./a,  encountered.  Rule  (b) 

l i 

with  k = 3 turned  out  to  be  preferable  to  (a). 

As  a byproduct  of  solving  (LKP),  Procedure  1 yields  a natural  candidate 
for  the  approximate  core  problem  (IP),  namely  the  one  defined  by  setting 


I 


if 


Kl  < 0 


otherwise 


where  Np  is  the  last  set  of  free  variables,  ordered  according  to  decreasing 

0 0 
c./a  ratios,  while  N is  an  (ordered)  subset  of  N such  that  In  I = 9 and 
1 i r r 1 F 1 

f,  the  index  of  the  fractional  component  of  x,  is  contained  in  the  middle 

0 

third  of  Np. 

The  size  of  I is  thus  controlled  by  the  parameter  0 , which  can  be 

chosen  equal  to,  or  slightly  greater  than,  the  desired  core  size.  However, 

the  actual  size  of  the  last  set  N may  be  considerably  smaller  than  0 . Also, 

F 

while  Np  is  guaranteed  to  contain  the  fractional  component  of  x,  this  compo- 
nent may  happen  to  be  closer  to  one  of  the  endpoints  of  the  interval  defined 
by  N than  to  its  center,  a configuration  which  does  not  make  for  a good 
approximation  to  the  core  problem.  When  one  of  these  situations  arises, 


/ 
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Procedure  1 "enlarges"  the  interval  defined  by  Np  in  the  desired  direction, 
by  adding  to  it  the  last  set  N> U N included  into  in  step  2 (if  the  enlarge- 
ment is  to  be  made  "to  the  left"),  or  the  last  set  N*^  U N_  included  into  NQ 
in  step  2 (if  one  wants  an  enlargement  "to  the  right").  These  sets  are  again 
natural  candidates  for  the  enlargement  of  I.  since  their  c^/a^  ratios  are 
closest  to  those  in  Np. 

Finally,  Prc cedure  1 also  produces  a first  integer  soli  t ion  x.  Namely, 
if  the  optimal  solution  found  for  (LKP)  is  x.  Procedure  1 sets  x = 1 if 
= 1,  Xf  = 0,  then  examines  each  remaining  variable  in  turn  and,  if 
possible,  sets  it  to  1. 

Next  we  shov'  that  the  computational  effort  involved  in  Procedure  1 is 
linear  in  n,  the  number  of  variables.  To  analyze  the  worst-case  bk uavior 
of  this  algorithm,  we  specify  the  choice  of  X in  step  1 as  that  of  the 
median  of  the  ratios  c./a.,  i eN  . Calculating  the  median  of  p numbers 

li  F 

requires  a computational  effort  linear  in  p;  more  precisely,  it  requires  at 
most  3p  comparisc ns  [11]. 

The  computational  effort  involved  in  Procedure  1 can  then  be  described 
as  follows. 

Preprocessing : n divisions,  to  calculate  the  ratios  c.^/a^. 

Step  1:  at  most  3 1 N_ I comparisons  to  calculate  the  median  of  the 

i jr  » 

ratios  c^/a^,  i e Np , plus  |n^|  comparisons  and  |Np|  additions  to  partition  Np 

and  calculate  S^(\),  S2(X.)>  i.c.,  a total  of  5 1 Np  | operations. 

Step  2:  two  comparisons,  plus  the  transfer  of  3 subsets  (which  involves 

changing  6 pointers),  i.e.,  8 operations. 

If  t is  the  number  of  iterations,  steps  1 and  2 are  executed  t times. 

If  the  algorithm  ends  at  step  2,  there  are  an  additional  | N^.  | comparisons 

and  IN  | additions  to  be  performed  in  order  to  assign  values  to  the  variables 
l p i 

V 1 6 v 
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Step  3:  this  requires  a number  of  at  most  c = O(81og29)  operations 

(sorting  at  most  0 elements,  plus'  assigning  values  to  the  corresponding 
variables).  This  number  is  independent  of  n. 

Since  | N^, | = n at  the  start,  and  is  reduced  by  at  least  at  eucli 

iteration,  at  the  end  of  the  procedure  |n  | < ~ n . 

* 2 

Let  w^  denote  the  time  needed  to  perform  a division,  and  the  time 
needed  for  a comparison,  addition  or  change  of  pointers.  Then  if  W(n) 
denotes  the  maximum  time  needed  to  solve  a problem  in  n variables  by 
Procedure  1,  we  have 

W(n)  < w,n  + w • 5n^  1 + 4 + ...  + -^r!+  8t+  2 • n + c 

— 1 2l  v 2 2t/  -> 

< w^n  + w2JJ5n^l  + + ...  + ■ + 8t  + cj 

< (w^  + 10w2)n  + 8w2log2n  + w2c. 

Hence  W(n)  = 0(n). 

A similar  analysis  of  the  expected  time  T(n)  needed  to  solve  a problem 
in  n variables  by  Procedure  1,  with  X chosen  randomly  in  step  1,  shows  that 
T(n)  is  given  by  an  expression  of  the  same  form  as  the  above  bound  for  W(n) , 
with  the  coefficient  of  n slightly  smaller  than  w^  + 10w2 . Computational 
experience  with  Procedure  1 is  to  be  found  in  section  6,  where  we  compare 
its  performance  to  that  of  two  sorting  routines  from  the  literature.  The 
comparison  is  unequivocally  favorable  to  Procedure  1. 

4.  A Heuristic  Whose  Accuracy  Improves 
Exponentially  With  Problem  Si2e 

Once  the  approximate  core  problem  (IP)  has  been  defined,  we  use  a 
heuristic  (Procedure  2 below)  to  find  a "good"  integer  solution  to  (IP). 
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Step  3:  this  requires  a number  of  at  most  c = O(01og20)  operations 

(sorting  at  most  0 elements,  plus  assigning  values  to  the  corresponding 

variables).  This  number  is  independent  of  n. 

Since  | N^,  | = n at  the  start,  and  is  reduced  by  at  least  at  eacli 

iteration,  at  the  end  of  the  procedure  | N | < ~ n . 

r 2 

Let  w^  denote  the  time  needed  to  perform  a division,  and  the  time 
needed  for  a comparison,  addition  or  change  of  pointers.  Then  if  W(n) 
denotes  the  maximum  time  needed  to  solve  a problem  in  n variables  by 
Procedure  1,  we  have 

W(n)  < w^n  + w^Sn^l  + + ...  + +8t+2*-^-n+c 

< Wln  + w2j  5n(l  + \ + . . . + + 8t  + c | 

< (w  ^ + 10w2)n  + 8w2log?n  + w2c. 

Hence  W(n)  = 0(n). 

A similar  analysis  of  the  expected  time  T(n)  needed  to  solve  a problem 
in  n variables  by  Procedure  1,  with  \ chosen  randomly  in  step  1,  shows  that 
T(n)  is  given  by  an  expression  of  the  same  form  as  the  above  bound  for  W(n) , 
with  the  coefficient  of  n slightly  smaller  than  w^  + 10w2_  Computational 
experience  with  Procedure  1 is  to  be  found  in  section  6,  where  we  compare 
its  performance  to  that  of  two  sorting  routines  from  the  literature.  The 
comparison  is  unequivocally  favorable  to  Procedure  1. 

4.  A Heuristic  Whose  Accuracy  Improves 
Kyuupnl  1 a I l_y  Willi  Jh'oli  I •*•••_ 

Once  the  approximate  core  problem  (IP)  has  been  defined,  we  use  a 
heuristic  (Procedure  2 below)  to  find  a "good"  integer  solution  to  (IP). 
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If  1^  was  exhausted,  stop;  otherwise  set  i ♦-  i+1  and  go  to  the  Iterative  Step. 

the  reason  for  not  setting  to  1 in  the  first  instance  those  such 

that  u^  - ak  < MIN,  is  that  doing  so  would  produce  an  integer  solution  which 

does  not  satisfy  the  knapsack  constraint  with  equality.  But  the  main  rationale 

behind  this  simple-minded  heuristic  is  that,  by  construction,  (IP)  has  all  of 

its  ratios  c^/a^  within  a small  range  around  the  ratio  c^/a^;  and  thus  an 

integer  solution  to  (IP)  which  satisfies  the  knapsack  constraint  with  equality 

is  likely  to  be  close  to  the  optimum,  and  so  is  the  associated  solution  to  (KP). 

Ci  Ci 

To  make  this  statement  more  precise,  let  r = max  — , r = min  — , and 

iel  ai  iel  ai 

u*  = max  u . Also,  let  v(KP)  and  v(LKP)  be,  as  before,  the  value  of  (an 
ieIl  1 

optimal  solution  to)  KP  and  LKP  respectively,  and  let  v be  the  value  of  the 

H 

solution  obtained  by  the  above  heuristic.  Then,  on  the  one  hand, 

v < v(KP)  < v(LKP) 
n — 

but  on  the  other, 

vR  > v(LKP)  - (r  - r)u* 

since  u^  is  an  upper  bound  on  the  sum  of  coefficients  a^  such  that  x^  = 1, 
x^  = 0,  and  r -r  is  an  upper  bound  on  the  change  in  the  objective  function 
,»alue  caused  by  a unit  change  in  that  sum.  From  the  last  inequality. 


hence  if 


v(LKP)  - vH  < (r  - r)u*  ; 


r-r  < — , 

u* 


then  v(LKP)-vH  < 1,  i.e.,  any  integer  solution  that  satisfies  the  knapsack 
constraint  with  equality,  is  optimal. 


I 


j 
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Let  x be  the  optimal  solution  to  (LKP)  found  by  Procedure  1,  let  xf  be 
its  fractional  component,  and  let  the  approximate  core  problem  be 


(IP) 

where 


max  { E c x£  | E a^  < aQ  , x£  - 0 or  1,  i c i]  , 
ieT  iel 


V1 


a°  = a°  " if1  ai 


while  I = {i1,...,it}  is  the  ordered  set  obtained  at  the  end  of  Procedure  1, 

which  contains  the  index  f of  the  fractional  variable.  Let  MAX  = max  a , 

iel  1 

MIN  = min  a , partition  the  ordered  set  I into 
iel 

ri  = and  T0  * » 


and  for  each  i el^,  define  u^  = a^  + a^x^. 

Since  the  knapsack  inequality  holds  with  equality  for  x,  u^  is  the 
"unfilled  capacity"  created  by  setting  x^-  and  x.  to  0. 

Procedure  2.  Set  = 0,  i «-  i^  and  go  to  the 

Iterative  Step.  Set  x^  = 0 and  try  to  match  the  "unfilled  capacity"  u^  with 
coefficients  a^,  k e Iq.  To  do  this,  consider  in  turn  each  k e Iq  and  set 

rk  = l»  ui  - ui  - ak  > if  ui  ■ V or  ui  - ak  ^ MIN 

% = °>  ui  *“  U1  » lf  ui  ■ ak  MIN  • 


If  as  a result  u.  = 0 is  obtained,  complete  the  solution  x by  setting 
x^  = for  all  components  whose  value  has  not  yet  been  assigned,  compare  x 
with  the  best  integer  solution  at  hand,  and  store  the  better  of  the  two. 

If  u^  = 0 is  not  obtained,  set  x^  = 1 for  the  first  k e Iq  such  that 
u^  - a^  < MIN;  then  complete  x and  proceed  as  above. 
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If  1^  was  exhausted,  stop;  otherwise  set  i *-  i + 1 and  go  to  the  Iterative  Step. 
The  reason  for  not  setting  to  1 in  the  first  inscance  those  Xj  such 
that  u -a  < MIN,  is  that  doing  so  would  produce  an  integer  solution  which 

1.  K 

does  not  satisfy  the  knapsack  constraint  with  equality.  But  the  main  rationale 
behind  this  simple-minded  heuristic  is  that,  by  construction,  (IP)  has  all  of 
its  ratios  c^/a^  within  a small  range  around  the  ratio  c^/a^;  and  thus  an 
integer  solution  to  (IP)  which  satisfies  the  knapsack  constraint  with  equality 

is  likely  to  be  close  to  the  optimum,  and  so  is  the  associated  solution  to  (KP). 

c c 

i i 

To  make  this  statement  more  precise,  let  r = max  — , r ■»  min  — , and 


iel  ai 


iel  ai 


u.  = max  u . Also,  let  v(KP)  and  v(LKP)  be,  as  before,  the  value  of  (an 

« I 

iel^ 

optimal  solution  to)  KP  and  LKP  respectively,  and  let  v be  the  value  of  the 

rl 

solution  obtained  by  the  above  heuristic.  Then,  on  the  one  hand, 


v < v(KP)  < v(LKP) 
H 


but  on  the  other, 

vH  > v(LKP)  - (r  -r)u^. 

since  is  an  upper  bound  on  the  sum  of  coefficients  a^  such  that  x^  = 1, 
x^  = 0,  and  r - r is  an  upper  bound  on  the  change  in  the  objective  function 
value  caused  by  a unit  change  In  dial  sum,  Prom  (lie  Iasi  Inequality, 

nl.M"  \ - v (v  - rlu^  ; 


hence  I I 


r 


- r 


1 


then  v(LKP)-v^  < 1,  i.e.,  any  integer  solution  that  satisfies  the  knapsack 
constraint  with  equality,  is  optimal. 
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Suppose  now  that  this  condition  is  satisfied,  i.e.,  the  difference  between 
the  best  and  worst  c^/a^  ratio  in  the  approximate  core  problem  is  less  than 
1/u^  . We  wish  to  examine  the  probability  that  Procedure  2 fails  to  find  an 
optimal  solution  to  (IP),  i.e.,  to  find  an  integer  solution  which  "fills  the 
knapsack". 

Procedure  2 systematically  tries  to  match  the  coefficients  a^,  k e 1^, 
against  the  values  u^  = a^  + af^f»  * e ^ the  coefficients  k € 1^  are 

uniformly  distributed  over  the  set  of  integers  between  MIN  and  MAX,  and  are 
independent  of  each  other,  the  number  of  possible  distinct  values  for  them 
is  Vq  = MAX  - MIN  + 1.  Let  denote  the  number  of  distinct  values  u^,  i € 1^, 

such  that  MIN  < u^  < MAX.  (By  definition  of  the  u^ , some  of  them  may  be 
outside  this  interval.)  Thus  we  have  a set  of  distinct  values  u^  from 
[MIN,  MAX],  and  a set  of  |Iq|  (not  necessarily  distinct)  values  a^,  uniformly 
distributed  over  that  same  interval.  Hie  probability  that  any  particular 
value  a^,  k e Iq,  does  not  match  any  of  the  V^  distinct  values  u;  is  1 - — . 

The  probability  that  none  of  the  | T q | values  a^,  kel^,  matches  any  of  the 

vx  I To I 

values  u^  is  then  I 1 - — ) . This  is  actually  an  overestimate  of  the 

\ vq  ' 

probability  of  the  failure  of  Procedure  2 to  "fill  the  knapsack,"  since  it 
ignores  the  possibility  of  each  u^ , i t 1^,  being  matched  by  the  sum  of  several 
Uk’  k e V rat^er  t*ian  by  a single  a^. 

The  remarkable  fact  about  this  simple-minded  heuristic  is  that,  if  the 
c^/a^  ratios  of  (IP)  are  sufficiently  close  to  each  other,  the  probability 

that  it  fails  to  find  an  optimal  solution  to  (IP)  decreases  exponentially 
with  |lg|>  which  is  more  or  less  the  same  as  exponentially  with  the  size  of 
(IP).  Values  of  the  above  expression  for  certain  values  of  its  parameters 
are  given  in  Table  1. 


V J^l 

Table  1.  Values  of  ^1  - ^ J 


VQ  = 50 


VQ  = 500 


.32  .11  .03  .01 

.16  .03  .004  .0008 


VQ  = 100 


10  30  50  70 


.82  .54  .36  .24 

.54  .15  .04  .01 

.35  .042  .005  .0006 


VQ  * 1,000 


\ig 

v\ 

10 

20 

30 

10 

.35 

.12 

.04 

15 

.20 

.03 

.007 

20 

.11 

.01 

.001 

25 

50 

75 

100 

.53 

oo 

CNJ 

• 

.15 

.08 

.28 

.08 

.02 

.006 

.14 

.02 

.003 

.0004 

Thus,  If  the  above  heuristic  is  applied  to  a knapsack  problem  with, 
say,  200  variables,  and  with  coefficients  randomly  drawn  from  the  interval 
[10,1000),  the  probability  of  finding  an  optimal  solution  is  about  .9996; 
and  the  larger  the  problem  (for  the  given  coefficient  interval),  the  higher 
this  probability. 
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A slightly  improved  version  of  the  heuristic  described  above  makes 
use  of  the  following  dominance  relation.  If  i,j  c N are  such  that 

Ci  2 ‘j  • "i-0)  • 

we  say  that  xi  dominates  x . If  at  least  one  of  the  two  inequalities  holds 
strictly,  we  say  that  x^  strictly  dominates  x^.  Clearly,  if  Xj  dominates 
Xj,  then  (KP)  has  an  optimal  solution  in  which  x^  > x^.  As  a result,  if  x^ 
is  set  to  0,  then  x^  can  also  be  set  to  0.  Incorporating  this  feature  in 
Procedure  2 improves  the  quality  of  the  solutions  generated,  though  for 
problems  with  a large  coefficient  range  the  effect  is  not  significant.  On 
the  other  hand,  for  problems  with  many  equal  c^/a^  ratios,  using  the  dominance 
relation  in  the  above  fashion  may  render  the  heuristic  ineffective,  by 
drastically  reducing  the  number  of  coefficients  a^,  k c 1^,  that  one  tries  to 
match  against  each  u^,  i c 1^.  This  effect  can  be  avoided  by  fixing  variables 
only  on  the  basis  of  strict  (rather  than  simple)  dominance. 


5.  Logical  Tests  and  Implicit  Enumeration 


The  sequence  of  Procedures  1 and  2 produces  an  integer  solution  x 
whose  quality  for  randomly  generated  problems  is  usually  very  good.  If 
the  rrnge  of  the  problem  coefficients  is  small  (say,  below  100),  then  the 
value  v of  the  integer  solution  x found  by  the  heuristic  of  section  4 often 
differs  by  less  than  1 from  v(LPK)  = cx,  in  which  case  of  course  x is  optimal. 
If  v(l,PK)  - v > 1,  we  apply  slightly  modified  versions  of  some  earlier  pro- 
cedures  from  the  literature. 

First,  we  use  some  simple  logical  tests  (Procedure  3)  to  fix  as  many  of 
the  out-of-core  variables  as  possible  (for  the  in-core  variables,  these  tests 
are  applied  simultaneously  with  the  heuristic  of  section  4).  The  first  of 
these  tests  (proposed  by  Dembo  and  Hammer  [6])  sets  permanently  x^  = if 
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the  reduced  cost  of  x^  matches  (or  exceeds)  the  above  gap;  i.e.,  If 


|c.  - a.(cf/af)|  > v(LKP)  - vR 


(where  xf  is,  as  before,  the  fractional  component  of  x).  The  second  test 

(proposed  by  Ingargiola  and  Korsh  [7])  sets  tentatively,  for  each  j in  turn, 

Xj  = l‘xj»  then  solves  the  linear  program  in  the  remaining  variables,  and 

if  the  value  of  the  solution  equals  or  exceeds  vR,  sets  permanently  x^  = x ^ . 

"Solving"  the  linear  programs  involved  in  this  test  amounts  to  the 

following.  Let  I = [i^,i2]  be  the  interval  defining  the  approximate  core 

problem  (IP),  and  for  i < i^,  denote  by  (LKP)x  _q  the  linear  program 

obtained  from  (LKP)  by  tentatively  setting  x^  = 1 - x^  = 0.  Then  we  set 

Xj=Xj,Vj<f,  j#i,  and  V j > and  then  "fill  the  knapsack"  by 

assigning  maximal  values  to  the  variables  x^,  f < J < i2»  taken  in  order 

of  increasing  j.  Whenever  this  is  possible,  the  resulting  solution  is 

obviously  optimal  for  (LKP)x  _R.  When  the  knapsack  cannot  be  "filled"  in 

this  way,  we  approximate  from  above  the  optimum  of  (LKP)  n by  introducing 

Xi 

a fictitious  variable  x^  whose  weight-coefficient  a^  "fills  the  knapsack" 

and  whose  cost  coefficient  satisfies  c /a  = c.  /a.  (where  i„  is  the 

a or  i2  i2  2 

last  index  of  I).  Such  an  overestimate  of  the  optimum  of  (LKP)  _ clearly 

xi“u 

preserves  the  validity  of  the  above  described  logical  test.  A perfectly 
analogous  procedure  is  used  when  one  tentatively  sets  x^  = 1 - x^  = 1 for 
some  i > i2# 

The  reason  for  applying  the  above  two  tests  in  this  particular  order  is 
that  the  second  test  is  more  powerful,  but  also  considerably  more  expensive 
computationally,  than  the  first  one.  It  makes  therefore  sense  to  apply  the 
second  test  to  the  reduced  problem  remaining  after  the  first  test  has  been 


used  to  fix  an  (often  significant)  number  of  variables. 
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The  power  of  the  second  test  can  be  enhanced  at  a low  cost  by  using 
the  dominance  concept  defined  in  section  4.  Thus,  whenever  x^  fails  the  test 
and  Is  therefore  (permanently)  set  to  I - x(  , we  also  set  to  l-ii^  all  the 
variables  dominated  by  xt  (il  l - x^  = 0),  or  all  the  variables  dominating  Xj 
(if  1 - x^  = 1).  The  reason  why  this  strengthens  the  procedure  is  that, 
contrary  to  intuition,  it  is  possible  for  a variable  x^  to  fail  the  test, 
and  for  a variable  Xj  dominating  or  dominated  by  x^ , to  pass  it. 

Procedure  3 consisting  of  the  above  tests  terminates  by  introducing  all 
the  variables  that  were  not  fixed  into  the  approximate  core  problem  (IP). 

If  this  redefined  problem  (IP)  has  more  than  s variables  (s  a parameter,  say 
s = 50),  then  the  heuristic  (Procedure  2)  is  again  applied  to  it,  followed 
by  the  logical  tests  (Procedure  3).  Otherwise  we  apply  the  final  part  of  the 
algorithm,  which  is  Procedure  4. 

Procedure  4 is  a virtually  unchanged  version  of  Zoltners'  implicit 
enumeration  procedure  [13],  called  by  its  author  a direct  descent  algorithm. 
(Another  name,  coined  by  Bradley  [5],  is  fixed  order  enumeration.)  The 
variables  are  ordered  according  to  decreasing  c^/a^  ratios.  At  each  node 
of  the  search  tree,  the  free  variables  are  scanned  in  the  above  order,  and 
all  those  that  "fit  into  the  knapsack"  are  set  to  1,  while  the  others  are 
set  to  0.  As  a result,  both  forward  steps  and  backtracks  usually  involve 
long  chains  of  variables,  i.e.,  many  levels  of  the  tree  (a  sequence  of  vari- 
ables fixed  at  0 can  only  be  "freed"  by  freeing  the  variable  fixed  at  1 
which  precedes  the  sequence).  For  details,  see  [13]. 

Hie  entire  algorithm  described  in  the  last  three  sections  is  summarized 
in  the  flowchart  of  Fig.  1. 

The  algorithm  discussed  here  is  highly  efficient  on  randomly  generated 
problems  (we  discuss  our  computational  experience  in  the  next  section).  An 
interesting  feature  of  our  approach,  however,  lies  in  the  fact  that  it  is  equally 
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well  (If  not  better)  suited  for  problems  whose  C|/fl|  ratios  are  very  close 
to  each  other.  A recent  survey  ol  the  0-1  knapsack  problem  by  K.  Martel lo  and 
P.  Toth  [9]  has  found  that  such  problems  tend  to  be  very  hard  even  for  the  most 
efficient  implicit  enumeration  procedures.  In  particular,  these  procedures 
tend  to  break  down  in  the  case  of  the  value-independent  knapsack  problem 
(i.e.,  the  problem  with  c^/a^  = const.,  Vi  eN),  mainly  because,  all  the 
reduced  costs  being  equal  to  0,  no  variable  can  be  fixed  by  the  logical  tests. 

As  to  our  algorithm.  Procedures  1 and  2 are  considerably  more  efficient 
on  (randomly  generated)  value-independent  knapsack  problems,  than  on  problems  / 

with  randomly  distributed  c^/w^  ratios.  Procedure  1 solves  (LKP)  in  one  step, 
since  the  first  A.  chosen  is  critical.  Procedure  2,  if  it  "fills  the  knapsack", 
finds  an  optimal  solution  which  is  also  instantly  proved  optimal,  since 
vH  = v(LPK). 

On  the  other  hand,  if  the  heuristic  does  not  "fill  the  knapsack",  then 
applying  the  logical  tests  of  Procedure  3 is  useless,  since  in  view  of  the  0 
reduced  costs  these  tests  must  fail.  However,  since  all  the  c^/a^  ratios 
are  equal,  the  probability  of  Procedure  2 failing  to  find  an  optimal  solution 
decreases  exponentially  with  the  size  of  (IP).  Therefore,  rather  than  applying 
Procedures  3 and  4,  the  thing  to  do  in  this  case  is  to  increase  (say,  double) 
the  size  of  (IP)  and  apply  again  Procedure  2.  This  can  be  repeated,  if 
necessary,  but  the  data  of  Table  1 show  how  low  the  probability  is  of  a need 
for  repetition. 

6,  Computational  Experience 

The  algorithm  described  in  the  previous  sections  was  coded  in 


FORTRAN  IV.  Efficient  handling  of  the  subsets  of  N and  easy  access  to  each 
of  those  subsets  was  accomplished  by  using  a linked  list  (circular  list) 


data  structure  of  the  kind  used  by  Barr  and  Ross  [4]  (and  also  Zoltners [13 ] ) . 

The  code  was  tested  on  200  problems  with  randomly  generated  objective  function 

and  constraint  coefficients,  and  with  aQ  “ ^ E a . To  examine  the  behavior 

1 eN 

of  the  algorithm  as  a function  of  both  problem  size  (number  of  variables)  and 
coefficient  range,  we  generated  20  classes  of  problems  (10  in  each  class), 
with  1000,  2500,  5000,  7500  and  10000  variables  respectively,  and  with  coef- 
ficients randomly  drawn  from  the  intervals  [10,50],  [10,100],  [10,1000]  and 
[10,10000]  respectively.  The  results  are  summarized  in  Table  2,  each  of 

whose  entries  gives  the  average  computing  time  for  10  problems  of  the  size  and  , 

characteristics  shown  by  the  respective  row  and  column  heading,  with  the 
exception  of  the  last  two  columns,  where  each  entry  gives  the  maximum  and 
minimum  time,  respectively,  used  for  any  one  problem  with  the  number  of  vari- 
ables indicated  by  the  row  heading,  irrespective  of  the  coefficient  range. 

Times  are  reported  in  UNIVAC  1108  milliseconds,  and  include  everything  except 
the  time  used  to  generate  the  data  and  print  out  the  results. 

The  times  reported  in  Table  1 were  obtained  with  a version  of  the  code 
which  conserves  memory  space  (a  scarce  commodity  on  C-MU's  UNIVAC  1108)  by 
assigning  only  4 integer  memory  units  to  each  variable  x^  (c^,  a^,  plus 
forward  and  backward  pointers).  In  particular,  the  c^/a^  ratios  are  not  stored 
in  this  version,  but  recalculated  each  time  they  are  needed.  For  some  smaller 

I 

problems  (1000  and  5000  variables)  we  have  also  tested  a version  of  the  code  in 
which  these  ratios  are  calculated  and  stored  when  the  data  are  generated,  and 
retrieved  as  they  are  needed.  The  execution  time  for  Procedure  1 (which  is 
between  40-60%  of  the  total  time)  is  then  reduced  to  about  35%  of  the  time 
needed  in  the  version  reported  in  Table  2 (see  last  column  of  Table  3). 

J 
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Table  2.  UNIVAC  1108  milliseconds  used  for  solving  (KP) 
(each  entry  Is  the  average  time  for  10  problems) 


A striking  feature  of  the  results  reported  in  Table  2 is  the  fact  that 
the  computational  effort  seems  to  grow  linearly  with  the  number  of  variables, 
and  logarithmically  (less  than  linearly)  with  the  range  of  the  problem  coef- 
ficients. Even  more  impressive  is  the  small  difference  between  the  average  and 
the  maximum  times  required  to  solve  the  40  problems  with  10000  variables  show  in 
the  last  row  of  the  tableau.  While  the  average  time  for  the  two  sets  of  10 
problems  with  coefficients  in  the  10-1000  and  in  the  10-10000  range  respec- 
tively, and  with  10000  variables  each,  is  about  2 seconds,  the  maximum  time 
for  any  of  the  40  problems  is  about  3 seconds  1 

To  assess  the  efficiency  of  the  various  components  of  our  algorithm 
separately,  we  generated  some  additional  data.  Table  3 below  compares  the 
time  needed  by  Procedure  1 to  solve  (LKP),  generate  a first  integer  solution 
and  define  the  approximate  core  problem  (IP),  with  the  time  needed  by  two 
sorting  routines  just  to  order  the  variables  according  to  decreasing  cVa^ 
ratios . 

The  first  of  these  sorting  routines,  SORT,  is  Singleton's  improved 
version  of  Quicksort  [121.  It  orders  an  array  of  integers  according  to 
decreasing  values.  The  time  used  by  SORT  to  accomplish  this  is  in  fact  an 
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underestimate  of  the  time  needed  for  preprocessing  the  data  in  order  to 
solve  (LKP)  in  the  traditional  way,  since  SORT  handles  at  each  step  only  one 
entry  for  each  i eN,  whereas  in  the  case  of  an  (LKP)  one  needs  to  handle  at 
least  4 entries  for  each  ieN:  c^,  a^,  c^/a^,  as  well  as  some  indication  of 

the  variable's  original  identity.  If  the  ratios  c^/a^  are  calculated  at 
each  comparison,  the  time  needed  for  sorting  is  even  larger. 

The  second  sorting  routine,  SORT  6,  is  the  one  used  by  Nauss  in  his 
algorithm  for  the  0-1  knapsack  problem  [10],  and  which  performs  all  the 

necessary  preprocessing  of  the  data  (it  calculates  and  stores  the  c^/a^  ratios,  / 

retrieving  them  as  needed).  Both  sorting  routines  were  used  as  originally 
coded  by  their  authors. 

Since  Procedure  1 accomplishes  more  than  SORT  and  is  therefore,  strictly 
speaking,  only  comparable  to  SORT  6,  we  also  ran  the  problems  with  1000  and 
5000  variables  respectively, with  a version  of  Procedure  1 (called  1*)  which 
solves  (LKP)  by  retrieving  the  pre-stored  c^/a^  ratios  as  needed. 

Each  entry  in  Table  3 represents  the  average  UNIVAC  1108  time  for  10 
problems  witli  coefficients  randomly  drawn  from  the  interval  [10,10000|. 

Times  are  in  milliseconds  and  include  everything  except  the  time  used  to 
generate  the  data  and  print  out  the  results. 

Table  3 speaks  for  itself.  Furthermore,  comparing  its  data  with  those 
ot  Table  shown  that  the  algorithm  dlucuHHod  In  thin  paper  solves  randomly 
generated  large  0-1  knapsack  problems  in  less  time  than  it  takes  to  order  the 
variables  according  to  decreasing  c^/a^  ratios. 

The  central  idea  of  our  approach  is  to  concentrate  on  the  core  of  the 
knapsack  problem,  approximated  by  (IP).  Thp  effectiveness  of  this  is  revealed 
dramatically  by  Table  4,  which  shows  the  results  of  constructing  the  approximate 


core  problem  (IP)  by  Procedure  1 and  applying  to  it  the  (heuristic)  Procedure  2, 
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Table  3.  UN1VAC  1108  milliseconds  used  for  sorting  the  variables 
(col.  1 and  2)  versus  solving  (LKP)  (col.  3 and  4) 

(each  entry  is  the  average  time  for  10  problems) 


Code 

No/'v\^ 

of 

variables 

1 

SORT 

2 

SORT  6 

3 4 

Procedure  I Procedur 

1000 

148 

536 

11.4 

37 

2500 

418 

1751 

270 

— 

5000 

878 

4163 

473 

166 

7500 

1377 

6206 

859 

— 

10000 

1855 

9645 

928 

— 

as  measured  by  the  quality  of  the  integer  solution  found  and  the  number  of 
variables  left.  This  table  refers  to  the  same  set  of  200  randomly  generated 
problems  as  Table  2.  Again,  each  entry  gives  the  average  for  10  problems. 

The  first  three  columns  refer  to  the  gap  between  v(LKP),  the  value  of  the  linear 
programming  optimum,  on  the  one  hand,  and  the.  value  of  the  first  integer  solution, 
produced  by  Procedure  I.  (col.  1),  the  value  v()  of  the  integer  solution  constructed 
by  the  (heuristic)  Procedure  2 (col.  2),  and  the  value  of  the  integer  optimum 
(col.  3),  on  the  other.  Column  4 gives  the  relative  error  of  the  integer 
solution  found  by  Procedure  2,  expressed  as  the  ratio  (v  -v(KP)/v(KP).  Column  3 
shows  the  number  of  variables  left  in  the  core  after  one  application  of  Proce- 
dure 2,  which  combines  the  heuristic  for  "solving”  (IP)  with  some  logical  tests 
for  fixing  variables  of  (IP).  Finally,  column  6 shows  the  number  of  out-of-core 
variables  left  after  Procedure  3,  which  uses  logical  tests  based  on  the  rela- 
tively narrow  gap  established  by  Procedures  1 and  2,  to  fix  as  many  variables 
as  possible. 


Tabl^  4.  Effect  of  "solving"  the  approximate  core  problem  (I?) 
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Table  4 shows  that,  given  a large  (KP),  solving  (LKP)  (without  sorting  the 
variables)  and  then  applying  the  heuristic  to  the  small  approximate  core  problem 
(20-50  variables)  generated  in  Lite  process,  produces  an  Integer  solution  within 
10  J — 10  ^ of  the  optimum,  and  makes  it  possible  to  lix  over  99 "L  of  the 
variables.  The  problem  whose  variables  have  to  be  sorted  in  order  to  apply  to 
it  the  implicit  enumeration  algorithm  of  Procedure  4 thus  has  typically  30-70 
variables . 

It  is  instructive  to  observe  the  effect  of  problem  size  and  coefficient 
range  on  the  gaps  listed  in  columns  1-3.  Each  of  the  3 gaps  seems  to  grow  linearly 
with  the  coefficient  range.  However,  while  the  first  gap  is  almost  unaffected 
by  problem  size  (number  of  variables),  the  other  two  gaps  tend  to  decrease 
with  problem  size.  For  the  third  gap,  i.e.,  that  between  v(LPK)  and  the  value 
of  the  integer  optimum,  this  Is  due  to  the  fact  that  for  a given  coefficient 
range,  an  increase  in  the  number  ol  variables  tends  to  increase  the  number  of 
Cj/Oj  ratios  which  are  close  to  Lho  critical  ratio,  and  thereby  to  Increase 
the  chances  for  the  existence  of  an  integer  solution  whose  value  is  "close" 
to  v(LKP).  In  other  words,  the  combinatorial  nature  of  the  problem  is  miti- 
gated by  an  increase  in  the  number  of  variables.  Similarly,  in  the  case  of 

the  second  gap,  i.e.,  the  one  between  v(LKP)  and  v , an  increase  in  the  number 

H 

of  variables  (for  a given  coefficient  range)  tends  to  reduce  the  range  of  the 
rai  los  spanned  by  the  approximate  core  problem  (IP),  which  in  turn  tends 

to  Increuse  the  accuracy  ol  the  heuristic  applied  to  (IP). 

7.  Hard  Problems 


The  ease  with  which  very  large  randomly  generated  0-1  knapsack  problems 
(including  value-independent  problems)  can  be  solved,  even  when  their  coef- 
ficients span  a sizeable  range,  is  somewhat  surprising  in  view  of  the  fact, 
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established  by  Karp  [ H ) , that  a polynomially  bounded  algorithm  for  this 
problem  can  only  exist  if  one  exists  for  the  general  integer  program.  This 
underscores  the  fact,  already  illuminated  by  the  similar  contrast  between 
worst-case  theoretical  bounds  and  statistical  behavior  in  the  case  of  the 
simplex  method,  that  often  there  exists  little  if  any  correlation  between 
the  worst  case  behavior  and  the  expected  behavior  of  algorithms.  In  partic- 
ular,  in  the  case  of  the  0-1  knapsack  problem,  the  worst  case  behavior  seems 
to  be  determined  by  a certain  special  structure  which,  in  randomly  generated 
problems,  arises  with  a very  low  probability. 

Since  we  have  found  that  those  0-1  knapsack  problems  usually  considered 
hard  (and  hard  they  are,  for  methods  based  on  logical  tests  and  implicit 
enumeration),  i.e.,  problems  whose  c /a  ratios  are  close  to  each  other,  are 
not  hard  at  all  for  our  approach,  we  set  out  to  explore  what  problems  are  hard 
to  hand le  by  our  method.  The  best  measure  of  the  degree  of  difficulty  of  a 
0-1  problem  that  we  could  find  was  the  relative  size  of  the  gap  between  the 
value  of  tiie  linear  programming  optimum  and  that  of  the  integer  optimum,  more 
precisely  the  ratio  between  this  gap,  v(I.KP)  -v(KP),  and  the  average  size  of 

• * | -/r 

the  reduced  costs  c.-a.A  , jeN  (here,  as  before,  \ - c./a.  is  the  critical 

1 j j 1 J f f 

cost/weight  ratio).  If  we  approximate  the  average  reduced  cost  bv  1/2  times  the 
largest  reduced  cost,  then  this  measure  of  the  degree  ot  difficulty  becomes 

A _ v(LKP)  - v (KP) 

' 1 I ‘ 

- max  |c  - a.\  | 

' UN  1 

Knapsack  pi  ob  loins  lend  to  have  a I at go  A it  there  are  relatively  few 


integer  solutions  which  satisfy  the  constraint  with  equality,  and  the  optimum 
is  not  among  them.  We  generated  a set  of  48  problems  with  this  property,  and 
found  that  only  about  1/4  of  them  could  bo  solved  easily  by  our  algorithm. 


r 
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On  the  other  hand,  we  have  tried  on  this  set  of  problems  a cutting 
plane  approach,  whose  efficiency  on  randomly  generated  problems  comes  nowhere 
near  that  of  enuinerative  procedures,  not  to  mention  the  algorithm  discussed 
in  this  paper.  Not  entirely  unexpectedly,  the  cutting  plane  approach  turned 
out  to  bo  relatively  efficient  on  the  "hard"  problems  with  large  A. 

The  cutting  planes  used  were  those  proposed  in  Theorem  2 of  [2).  Iliese 
inequalities  are  computationally  cheap  and  in  most  cases  (statistically,  in  more 
than  3/4  of  the  cases)  are  facets  of  the  knapsack  polytope,  i.e.,  of  the 
convex  hull  of  feasible  0-1  points.  When  they  are  not  facets,  their  coeffi- 
cients are  known  [3]  to  differ  by  at  most  l from  the  corresponding  coefficient 
of  a facet  (with  the  same  right  hand  side). 

The  algorithm  we  used  is  as  follows.  First,  Procedures  1,  2 and  3 of 
the  previous  sections  are  used  to  solve  (LKP),  find  a good  integer  solution 
to  (IP),  and  fix  as  many  variables  of  (KP)  as  possible.  If  the  problem  in 
the  remaining  variables  is  sufficiently  small,  we  solve  it  by  Procedure  4 
above.  Otherwise,  we  generate  an  inequality  which  cuts  off  x,  the  linear 
programming  optimum,  and  solve  the  2-constraint  linear  program  (with  the 
knapsack- inequa li ty  and  the  cut).  The  knapsack  inequality  is  then  replaced 
by  a combination  of  the  two  constraints  (with  the  optimal  dual  multipliers 
used  as  weights),  which  produces  a new  knapsack  problem,  with  a smaller  value 
of  the  linear  program  and  hence  with  a smaller  gap.  We  then  apply  Procedures  2 
and  3 to  this  new  knapsack  problem,  in  which  the  smaller  gap  often  leads  to 
the  fixing  of  new  variables.  The  above  sequence  could  be  iterated  for  as  long 
as  a decrease  in  the  gap  can  be  obtained;  but  we  stopped  after  the  first  cut, 
since  the  purpose  of  this  preliminary  experiment  was  to  get  a feel  for  the 
merits  of  the  cutting  plane  approach  in  general. 
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We  have  tested  this  approach  on  48  problems  of  the  type  described  above, 
with  1000  variables  each,  and  with  coefficients  in  the  range  1-100.  Seven  of 
the  48  problems  were  solved  (optimality  was  proved)  by  Procedures  1 and  2. 

Another  5 problems  were  reduced  by  Procedures  1-2-3  to  a sufficiently  small 
size  to  be  easily  solved  by  Procedure  4.  This  makes  for  a total  of  12  problems 
solved  by  the  algorithm  of  sections  3-5.  Of  the  remaining  36  problems,  whose 
average  A was  .076,  18  were  solved  by  the  cut  (optimality  was  established  due 
to  the  tighter  bound  obtained  by  solving  the  2-constraint  linear  program), 

while  7 were  reduced  to  a size  which  made  it  easy  to  solve  them  by  Procedure  4.  t 

In  all  these  25  problems  the  optimal  solution  was  generated  by  the  heuristic 
(Procedure  2),  and  the  role  of  the  cut  was  to  make  it  possible  to  prove 
optimality.  For  the  remaining  11  problems,  the  cut  had  practically  no  effect. 
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j We  describe  an  algorithm  for  the  0-1  knapsack  problem  (KP),„  which  relies 

mainly  on  three  new  Ideas.  The  first  one  Is  to  focus  on  what  we  cal ^ the  ( 
j core  of  the  problem,  namely  n knapsack  problem  equivalent  to  (KP1,  defined  I 

on  a particular  subset  of  the  variables.  The  size  of  this  core  Is  usually 
\ a small  fraction  of  the  full  problem  size,  and  does  not  seem  to  increase 

with  tbe  latter.  While  the  core  cannot  be  Identified  without  solving  (KP1 , > 

> a satisfactory  approximation  can  be  found  by  solving  (LKP),  tbe  associated 
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• linear  program.  The  second  new  Ingredlenl  Is  a hinary-search-type 
» procedure  for  solving  (LKP)  which,  unlike  earlier  methods,  does  not 
i require  any  ordering  of  the  variables.  \ The  computational  effort  Involved 
, in  this  procedure  is  linear  in  the  number  of  variables. 3 Finally,  the 
j third  new  feature  is  a simple-minded  heuristic  whose  accuracy  under 
| certain  conditions  grows  exponentially  with  the  problem  size. 

Computational  experience  with  an  algorithm  based  on  the  above  ideas, 
on  200  randomly  generated  test  problems  with  1,000-10,000  variables 
! and  with  coefficients  ranging  from  between  10-100  to  be  between 
‘ 10-10,000,  indicates  that  for  such  problems  the  computational  effort 

grows  linearly  with  the  number  of  variables  and  logarithmically  with 
the  range  of  coefficients.  Total  time  for  the  200  problems  was  160 
UNIVAC  1108  seconds,  and  the  maximum  time  for  any  single  problem  was 
3 seconds. 
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